A problem to be addressed is light deficient photography in the presence of motion during image capture causing motion blur in the image. The motion can be of a global variety where the entire scene being imaged moves together or of a local variety where one or more portions of the scene move at a different speed or direction compared to the rest of the scene. Global motion is due to a relative motion between the scene and the camera during image capture. Local motion is due to objects within the scene moving relative to the other portions of the scene. Local motion can occur differently in different portions of the scene.
In the case where the exposure time is short and motion is slow, a single image capture can be used to capture an image with good image quality. However, as the exposure time required to get an image with a high signal-to-noise ratio becomes longer relative to the motion that is present during the image capture, or the available light decreases, captured image quality degrades in the form of increased motion blur and increased noise within the image.
For consumer digital cameras, there is a trend for the size of the pixels to get smaller over time, which decreases the available area to capture light during the exposure, so that noise becomes more of a problem. Exposure times are increased to compensate for smaller pixels, but then motion blur becomes more of a problem. Consequently, methods to increase the sensitivity of pixels to light have been described as in U.S. Patent Publication No. 2007/0024931 by Compton, et al, which adds panchromatic pixels to the image sensor.
As the motion increases relative to the exposure time or the available light decreases there are a series of compromises that are made. Several compromises are made in photographic space to increase spatial resolution, temporal resolution or image quality, but, it is important to note that with each gain there will also be losses. For instance, the exposure time is decreased to reduce motion blur thereby increasing temporal resolution, but at the expense of increasing noise in the image.
Flash is an effective way to reduce noise in the image and by enabling a shorter exposure time, the temporal resolution is increased but at the expense of uneven lighting and redeye. Additionally, there are image capture situations for which a flash is either not available or not beneficial.
Optically based image stabilization is used during the exposure to enable a longer exposure time to reduce noise in the image while decreasing motion blur and increasing spatial resolution. However, optically based image stabilization can only be used to reduce motion blur from camera motion (global motion). Additionally, optical image stabilization increases the cost and weight associated with the image capture device.
The effective size of the pixels is increased to enable a reduced exposure time by binning (i.e., adjacent pixels are connected to one another so the charge on the adjacent pixels is summed and the signal is increased). However, binning is accompanied by a decrease in spatial resolution.
Another method to reduce blur is to capture two high resolution images, one with a short exposure time, and one with a long exposure time. The short exposure time is selected so as to generate an image that is noisy, but relatively free of motion blur. The long exposure time is selected so as to generate an image that has little noise, but that can have significant motion blur. Image processing algorithms are used to combine the two captures into one final output image. Such approaches are described in U.S. Pat. No. 7,239,342, U.S. Patent Publication No. 2006/0017837, U.S. Patent Publication 2006/0187308 and U.S. Patent Application Publication 2007/0223831. The drawbacks of these approaches include a requirement for additional buffer memory to store multiple high resolution images, additional complexity to process multiple high resolution images and potential gaps in time between the two image captures.
Another method to reduce blur is through image restoration—de-blurring—algorithms applied post-capture. An example of such an approach is the well-known Lucy-Richardson de-convolution algorithm. Drawbacks of this and similar approaches include high computational complexity and sensitivity to noise.
Multiple low resolution video images can be used to form a single image with improved image quality while maintaining spatial resolution and offering a balance between temporal resolution and exposure time. Reading multiple images within a given time can reduce motion blur by using a shorter exposure time for each image, however, each image will be noisier. By aligning the multiple images with each other to correct for motion between individual image captures and then summing the individual images together, the noise is reduced in the formed single image.
While multiple low resolution video images are read out relatively quickly (30-60 images/sec is typical), and the images typically have lower noise since the pixels are often binned, the single image that is formed is limited to relatively low resolution. Conversely, multiple high resolution images are used to form a high resolution single image. However, high resolution images typically are noisier since the pixels are smaller and more significantly, a relatively large amount of time is required to readout multiple high resolution images (1.5-7 images/sec is typical) due to hardware limitations. In addition, the problem of aligning the images grows large due to significant motion between the image captures.
Therefore, a need in the art exists for an improved solution to combining multiple images to form an improved image, especially in scenes where motion is present.